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Abstract 

We present a series of musical instrument prototypes that 
inquire about how live music can be produced simultane- 
ously in the tangible world and within the metaverse. Our 
study is centred around a workshop with seven young Inter- 
action Design students and follows a Research-through-De- 
sign approach. We invited participants to ask what musical 
instruments for the metaverse should look like? how they 
are played? and what affordances do they provide for both 
tangible and virtual audiences and performers? Participants 
prototyped working electronic musical instruments that can 
be played and listened to in both the real world and in virtu- 
al space. We took a Critical Design stance and followed a vi- 
sion-driven design process, foregrounding the non-existent 
rather than problem-solving or market demands. 


The resulting physical prototypes deviate from existing mu- 
sical instruments and offer novel interactions that affect 
how music is performed and listened to. Our findings show 
that hybrid music-making provides possibilities for collab- 
oration, explorations of scale between tangible and virtual 
worlds, and notions that challenge who is in control of the 
music-making process. We also analyse how mapping be- 
tween senses and feedback drive experiences for perform- 
ers and audiences alike. 
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Introduction 

The digitisation of sound dramatically transformed music, 
its production, distribution, and consumption. Since the dis- 
appearance of the mix-tape in favour of mp3 players and 
the usage of samples from online files in live performanc- 
es, digitality has blurred the roles of musicians, distributors 
and consumers (De Notaris & Savonardo, 2022). As Internet 
connections grew faster, mobile and broadly available our 
lives became mediated by online services and social media 
platforms. Our tangible world is increasingly interwoven with 
software and the idea of inhabiting virtual environments -the 
metaverse- has gained momentum with attempts to turn it 
into reality (Ball, 2021; van der Merwe, 2021). In this work we 
explore how music can simultaneously be performed and lis- 


tened to both in-person and within the metaverse. We focus 
on musical instruments as tangible and virtual objects that 
emit sound in both of these environments. 


With the metaverse beginning to be recognized as a space for 
musical opportunity, we ask ourselves how live music can be 
performed and experienced within it. There has been some 
research around networked music performance (e.g. (Rotton- 
di et al, 2016)) and some companies (e.g. Sensorium Galaxy, 
Mizic.io) are starting to offer virtual reality (VR) and hybrid 
concerts, with some of these happening within video game 
environments (Groux, 2020). Attention has also been placed 
on how networked music enables collaboration between 
musicians and enhances audience participation (Carot et al., 
2009; Xambó et al., 2017; Yamchareon & Herkenrath, 2005). 
These studies focus mostly on the technicalities of synchroni- 
zation or on the social aspects of collaboration and how they 
exist online. There is however little work around the design of 
musical instruments for online and VR performance with Ser- 
afin and colleagues (Serafin et al., 2016) and Turchet (Turchet, 
2019) presenting some ideas and guidelines. The instruments 
described for VR and networked performances in these works 
resemble traditional instruments such as guitars, xyloghones 
and assorted percussion instruments. We therefore centre our 
research on what novel instruments for hybrid performance 
look like, how they are played, and how they sound; trying to 
distance our work from existing instruments. 


Focusing on instruments as objects allowed us to follow a 
Research-Through-Design methodology (RtD). Taking ideas 
from Frayling (Frayling, 1993), Gaver (Gaver, 2012), Anders- 
en (Andersen, 2014) and, Wensveen and Matthews (Stephan 
Wensveen & Ben Matthews, 2015) we consider the process 
of discussing and developing prototypes as a form of inquiry. 
Since our research questions revolve around objects which 
do not yet exist we find Critical Design (Dunne & Raby, 2001) 
to be a useful frame for our design activities as its vision-driv- 
en approach allowed us to consider possibilities that are nei- 
ther problem-solving nor market-driven. 


Metaphysical instruments 

Using prototypes as objects of inquiry invited us to write a 
brief guiding the research and development of musical in- 
struments for hybrid performance. We named the brief Meta- 


physical Instruments, a combination of the words Metaverse 
and physical that also references philosophical metaphysics. 
Metaphysics’ ideas of being were useful for exploring how re- 
ality and experience differ within the Metaverse and outside, 
however our engagement with metaphysics was superficial 
as an in-depth analysis was beyond the scope of this work. 


Our brief considered the divergent experiences of performing 
and listening to live music within the metaverse versus tangi- 
ble space and prompted the following questions: How should 
instruments for hybrid performance be played? how should 
they sound? What social opportunities are there for live mu- 
sic in VR? And what novel experiences can VR provide for live 
music? These questions were broad and ambiguous allowing 
participants to project and use their own definitions of music, 
instruments, and live performance. 


Workshop 

Given the complex skillset involved in making electronics and 
VR we took inspiration from Bowers and colleagues (Bowers 
et al., 2016) in organizing an open-ended workshop that lever- 
aged participants’ existing knowledge. We invited interaction 
design students to a 12-week workshop. Seven undergradu- 
ate students (authors four to eleven) joined and divided them- 
selves into three groups. Participants were in the second year 
of the Design for Digital Media and Technology programme at 
CENTRO University and had knowledge in programming, 3D 
modelling, animation and digital drawing. The first four weeks 
of the workshop were spent developing skills in electronics, 
microcontroller programming (ESP-32 and the Arduino IDE), 
digital manufacturing (8D printing, CNC mills and lasers) and 
VR development using the Unity game engine. The final eight 
weeks were devoted to prototype production. 


The workshop started with a launch presentation explaining 
the brief and an outline of existing experimental musical in- 
struments was provided. We wanted to distance ourselves 
from stablished categories of instruments such as (Temilola, 
2020; Turchet et al, 2022) and encourage participants’ defi- 
nitions of what an instrument is and can be. Following our 
Critical Design perspective they were also introduced to vi- 
sion-driven design processes. A short description of each of 
the resulting prototypes follows. 


CQNC: Community in Quest for Noise Connection 
Patricio Pous-Pierson, Alejandro Lobo-Barrera, and Jordi Fra- 
goso-Terreros developed CQNC (pronounced sequence). It 
bridges the metaverse and tangible reality through real-time 
sound recording. CQNC uses field recordings to produce mu- 
sic and explores how these can work as an instrument. It re- 
quires at least two players, one using a VR headset while the 
other physically carries a device (Fig.1, left) capable of record- 
ing sounds and uploading them to a server. 


When a recording is made it is downloaded into VR and repre- 
sented as a ball. Users can then place sounds into a playback 
area (Fig.1, right). Sounds in this area are looped and mixed into a 
single output. A sound's volume is modified by moving it up and 
down in space and its playback speed and pitch are controlled by 
rotating the sphere. New recordings increase the samples avail- 
able for mixing. In VR players compose complex soundscapes 
from samples recorded live in distant environments. 
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Figure 1. The CQNC sound recording controller (left) and its VR environment (right) 
showing the playback area and the sounds it contains. 
Video documentation can be found at https://en.centro.edu.mx/3JC4yd1 


Conditioned Freedom (CF) 

In CF Paola Ferrari-Garcia and Ximena Pefa-Rios present an 
interactive sound installation. It provides contrasting experi- 
ences for tangible and VR users by placing one of them with- 
in a woven structure (Fig 2, left) filled with light beams and 
annoying sounds. Blocking a light mutes one of the sound 
channels and makes the space more bearable. Within VR us- 
ers experience an open landscape (Fig 2, right), an opposite 
environment to that of the physical installation. Each time a 
light is covered in the installation a relaxing sound is played in 
VR accompanied by a visual effect. 


Figure 2. The Conditional Freedom installation (left) within which the performer 
interacts with light to control sounds both in tangible space and within 
a VR landscape (right) 


CF's driving concept was that art is experienced through its 
end result but we seldom see the struggle of making it. Their 
title links this struggle with constrains in creative freedom 
and places the physical user in the role of a creator and the 
VR audience as spectators of a finished work. In their view the 
greater the struggles in the tangible world the more beauty is 
produced in VR. 


Llum 

The instrument made by Fabiola Toledo-Galindo and Julio Tor- 
res-Cazares is played and listened to both from the outside 
and from within. Llum is a handheld tangible instrument and 
a large virtual space at the same time. The project explores 
how the functionality of a tangible object can change when 
its scale is altered. 


The instrument is a 3D-printed cracked sphere that emits 
light from within (Fig 3, left). Each piece works as a key that 
synthesizes sound and light in both tangible space and VR. Its 
virtual counterpart is an enlarged version of the same sphere 
(Fig 3, right) within which the VR audience exists. The walls of 


this space match the object's keys in shape and functionality; 
That is, if a user touches a wall, it will synthesize the match- 
ing sound both physically and virtually. The nature of the in- 
strument blurs the line between audience and performer as 
they are all capable of contributing to the composition. For 
this collaboration to work sounds from each key are added 
together when they are played simultaneously, either physi- 
cally or virtually. The sounds produced in VR therefore match 
those emitted by the instrument in tangible space. 


Figure 2. Llum, an instrument that is played both physically (left) and virtually (right). 
VR users experience and play the instrument form the inside while tangible 
ones hear the sound coming from within the object. Video documentation 


can be found at https://en.centro.edu.mx/3YSY65z 


Findings 

The following subsections analyse how the prototypes func- 
tion as musical instruments, how they approach liveness, and 
how scale and control were used in driving users’ experiences. 


Mapping and feedback 

Serafin and colleagues (Serafin et al., 2016) highlight the im- 
portance of feedback and a clear mapping between senses in 
VR musical instruments. We base our initial analysis on these 
features. Both Llum and CF map gestures to sound with visual 
and auditory feedback. In Llum’s case, the single sound out- 
put that amalgamates inputs from all participants breaks the 
auditory feedback, as performers increase in number it be- 
comes difficult to recognize sounds triggered by individual 
users. This collective feedback makes players respond to one 
another's actions rather than focusing on individual composi- 
tion, turning the metaverse and live-music into social spaces. 
This suggests that communal feedback can favour collabora- 
tive and improvisational practices. 


The CF installation also has a direct mapping between visi- 
ble rays of light and sound but contrary to other instruments, 
gestures remove sound rather than creating it. The sounds 
produced in VR do not provide feedback to the performer to 
whom the resulting composition is unknown. The contradic- 
tory interactions arising from this multiplicity of experiences 
resonate with Dunne’s view that objects working contrary to 
established logic can be very engaging (Dunne, 2008). In CF 
the instrument's feedback also takes a narrative role in a sto- 
ry, the performer's attempts to silence the whole installation 
are contextualised as a creative struggle. 


CQNC challenges views of what an instrument and its mu- 
sic are. There has been research presenting field recordings 
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as music (Shaw & Bowers, 2020). However, CQNC provides 
a novel perspective as it is difficult to decide what the in- 
strument is: Is it the sound recorder as an object or is it the 
spheres sounding in virtual reality? This difficulty in assessing 
where the music-making happens and who the performer is 
suggests that it is truly collaborative and explores musical- 
ity in a way only made possible through networked co-cre- 
ation. Referring back at Serafin and his team’s principles for 
VR instruments we find that CQNC breaks some of these: it 
has no direct feedback or mapping, it embraces latency and 
provides no visualization of a player's body. It is yet engaging 
and rewarding to use, suggesting that current definitions of 
instruments for VR and the metaverse should be questioned 
in the field's early days. 


Virtual and Live 

Liveness is central to the resulting prototypes. Not only in 
the technicalities of device-metaverse communication, but 
also in overcoming the lacking senses of unity and unexpect- 
edness inherent to remote emulations of live-music as de- 
scribed by Tarumi an co-authors (Tarumi et al., 2017). 


Llum’s collaborative nature demands that tangible performer 
and VR audience co-exist. This temporality gains depth when 
seen from a social perspective. As the virtual audience shares 
the same space, they must also synchronise their actions in 
order to drive their portion of the composition. We believe 
that implementing communication between audience mem- 
bers in a future version would further enhance this social co- 
ordination. 


In CQNC liveness is less straightforward. Tim Shaw describes 
field recording as “a practical activity that often requires one 
to spend long periods of time outdoors hunting for sound” 
(Shaw & Bowers, 2020). This is seemingly contrary to live per- 
formance, waiting for sound samples to arrive could frustrate 
players in VR. However, receiving a new sample provides an 
exciting element of unexpectedness. Users were found to re- 
start compositions when a good sample appeared. Simulat- 
ing this with pre-recorded sounds could be done but knowing 
that they are being recorded live adds a sense of complicity 
between tangible and virtual players. 


Control 

The prototypes revealed issues of power and control that 
were not built-in by design. By control we mean people having 
power over one another and not interface features. In most 
performances musicians have complete control over what 
to play, how to play it and when. Our work suggests that the 
distribution of control is central to the metaverse but difficult 
to define. 


In CF control is approached from an unusual angle, the person 
within the installation has complete control over sound pro- 
duction but that control is unpleasant to wield and must be 
endured for the pleasure of the virtual audience. This creates 
a dynamic in which the person in the installation controls 
the content but is emotionally subordinated to those in the 
metaverse. 


CQNC and Llum show a different form of control, one in which 
users react to sounds provided by others. These instruments 


allow adversarial or collaborative interactions between par- 
ticipants, where players will conform to what is being pro- 
vided or try and subvert it. Llum distributes control amongst 
participants: If few people are attending in VR control shifts 
towards the person playing physically. In contrast, with tens 
of VR participants, the notes from the tangible instrument 
are quickly outnumbered by collective online behaviours. The 
distribution of control in CQNC is slightly simpler, every play- 
er ultimately controls their own composition but VR players 
have no agency over the received sound and must work with- 
in the constraints created by the person recording. It is also 
possible that the recorder feels pressure to provide adequate 
samples, despite not knowing what each player is doing, re- 
sulting in a self-constrained curation of sounds. 


Scale 

All three prototypes incorporate aspects of scale. The digital 
flexibility of scale has been extensively explored within design; 
however, we see value in presenting it from a music-making 
perspective. Llum incorporates shifts in scale by making the 
tangible instrument inhabitable within VR space. This shows 
that VR can not only place users in otherwise inaccessible 
spaces but also alter our reading of tangible objects: The in- 
strument ceases to be to a standalone object and becomes a 
model of the VR space, therefore containing its audience. This 
illusion of containment is further reinforced by making the 
tangible instrument emit sound and light from within while in 
VR it is spatially emitted from the key that was pressed. 


CF uses scale to drive its concepts. The installation as a con- 
strained space is designed to overwhelm the user. Its virtu- 
al counterpart is a large, well lit, and calming landscape. The 
interplay of scales between VR and installation provides an 
entry-point to the project's ideas, appreciating this difference 
in scale is sufficient to understand the project. 
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Both CF and Llum present VR worlds much larger than their 
physical counterparts. Contrastingly, in CQNC the scale of VR 
space is fairly small, with minimal movement and interac- 
tions restricted to a sphere in the middle. This limited space 
serves as an interface for an instrument that could be of 
global scale. Reducing sounds gathered over a large area to 
such a small space suggests that the metaverse could be not 
only an alternative world but also a container for parts of our 
tangible reality. 


Conclusion 

Designing, developing and using our musical instruments 
provoked many questions that were not originally foreseen. 
We set out to make prototypes that questioned the ontology 
of instruments in the metaverse, outlining what they are and 
how they exist. This starting point was however greatly ex- 
panded through our process. 


Our findings around feedback and mapping do contribute to 
characterising our prototypes as instruments but they also 
promted questions about collaboration and improvisation 
that emerge from the instruments’ online nature. When dis- 
cussing liveness, we were surprised to be focusing on feelings 
of unity and the unexpected rather than the technicalities of 
internet communication protocols. The themes of scale and 
control were not initially identified in our research and were 
prompted by the prototypes, we find them very relevant 
when discussing the metaverse and see their emergence as 
an indicator of how open-ended and protype driven process- 
es can elucidate unidentified questions that contribute to re- 
search. We would like to expand on this inquiry in future work 
by further exploring our prototypes through live performanc- 
es with a breath of different audiences and performers from 
various musical backgrounds. 
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